There is an ever increasing demand for building larger storage systems, driven by primary data growth and by the advent of new workloads such as disk-based backup. Backups which were traditionally stored on tapes are now being stored on disk-based storage systems for better performance and cost effectiveness. Such backup systems have huge footprints often several times larger than traditional primary storage systems and yet are unable to meet the requirements of the biggest enterprise customers.
Unfortunately, scaling file system capacity is not a trivial problem particularly for de-duplicated systems because of their huge memory requirements. De-duplicated file systems maintain some metadata in memory for every block of physical data. This metadata overhead ranges from few bits per block for optimized file systems to over a dozen bytes for more rudimentary implementations. It is not trivial to add memory to a system to scale with capacity. Traditional server class motherboards have limits on the amount of memory that can be attached to them. Main memory is also a “premium” resource; not only is it one of the most expensive components in the system, it is often one of the most energy consuming one. Hence there is always the need to reduce the memory footprint of a file system.